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DETAILED ACTION 

Response to Amendment 

This Office Action is responsive to the Amendment filed July 1 , 2008. Claims 1 , 
4-6,8, 11-13, 15, 21, 24, 27, and 31 are currently amended. Claims 3, 10, 23, 26, and 
29 are cancelled. Claims 1,8,15,21, and 27 are independent. Claims 1 , 2, 4-9, 1 1 -22, 
24, 25, 27, 28, and 30-34 remain pending. 



Claim Rejections - 35 USC § 102 

The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by 
another filed in the United States before the invention by the applicant for patent or (2) a patent 
granted on an application for patent by another filed in the United States before the invention by the 
applicant for patent, except that an international application filed under the treaty defined in section 
351(a) shall have the effects for purposes of this subsection of an application filed in the United States 
only if the international application designated the United States and was published under Article 21(2) 
of such treaty in the English language. 

Claims 1,2,4, 6-9, 11,1 3-22, 24, 25, 27, and 28 are rejected under 35 
U.S.C. 102(e) as being anticipated by Gokturk et al. (hereinafter Gokturk), United States 
Patent Application Publication 2003/0235341. 

Regarding claim 1, Gokturk teaches a system for processing captured 
multimedia information for insertion into a communication using an Instant Messaging 
(IM) application (see paragraph [0079]; "Using the current invention, one can design an 



IM application where the user's facial impression is translated to an avatar and 
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transmitted as part of the user message"), wherein the insertion is based on multimedia 
information, the system comprising: 

an information capture module for capturing the multimedia information in the 
vicinity of a machine on which the user is using the IM application (see Figure 8a and 
paragraph [0074]; "Camera 810 captures the user image in real-time"); 

a processing module communicatively coupled with the information capture 
module, for processing the captured multimedia information in real time to create a 
graphic (see Gokturk Figure 8a and paragraph [0074]; "Camera 810 captures the user 
image in real-time"; see also Gokturk paragraph [0078]; "Identifying objects in an image 
is the first step in performing more comprehensive image processing to determine 
identity of a user or detect facial impressions such as happy face, sad face, puzzled 
face, etc"); and 

an Application Program Interface module for the IM application, communicatively 
coupled to the processing module, for inserting the graphic into the communication in 
real time using the IM application, said inserting only occurring after detecting a trigger 
from a user (see Gokturk paragraph [0079]; "Using the current invention, one can 
design an IM application where the user's facial impression is translated to an avatar 
and transmitted as part of the user message"; see also Gokturk paragraph [0074]; "To 
conserve bandwidth, the server may choose to stream the video of select users among 
the participants in the forum (e.g. the person who is typing)."). 
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Regarding claim 2, Gokturk teaches that the multimedia information comprises 
at least one of audio information, still image information, and video information (see 
paragraph [0079]; "the camera captures the user image"). 

Regarding claim 4, Gokturk teaches that the graphic is one of a low resolution 
image with accentuated expression information, an animation and a stylized version of a 
person's face (see paragraph [0079]; "In the receiving computer, a table is maintained 
that maps the code to the stored image of the avatar. This table is used to map the code 
back to the avatar representing the facial impression of the sender. Instead of a code 
word, the application may choose to send the image of the avatar"). 

Regarding claim 6, Gokturk teaches that the graphic is an emoticon (see 
paragraph [0078]; "Avatars are symbols that are typically interspersed with text as a 
short cut symbol or to spice up the text for conveying, say, user emotion"). 

Regarding claim 7, Gokturk teaches that the graphic is created by the user by 
processing captured multimedia information (see paragraph [0079]; "the camera 
captures the user image. By performing an image-processing algorithm, either on the 
local computer or a connected server (by first sending the image to server), a computer 
program analyses the facial impression of the user. The computer program maps the 
user impression to one of the predefined facial emotions represented with 
corresponding avatar"). 

Claims 8, 9, 11, 13, and 14 recite a method having substantially the same 
limitations as the system of claims 1 , 2, 4, 6, and 7, respectively. Therefore, the claims 
are rejected under the same rationale. It should be noted that claims 11-13 refer to a 
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plurality of graphics as opposed to a single graphic, which Gokturk also anticipates in 
paragraph [0079]. 

Claims 15 and 16 recite a method having substantially the same limitations as 
the system of claims 1 and 3, respectively. Therefore, the claims are rejected under the 
same rationale. 

Regarding claim 17, Gokturk teaches storing the graphic for use in a later IM 
communication using the application (see paragraph [0079]; "In the receiving computer, 
a table is maintained that maps the code to the stored image of the avatar"). 

Regarding claim 18, Gokturk teaches that the step of processing the received 
captured multimedia information to create a graphic comprises: reducing the size of the 
captured multimedia information (see paragraph [0075]; "Instead of transmitting the 
entire image, only the relevant segment (in this case, the foreground picture of the user) 
is compressed and transmitted. For instance, if the size of the segment were 50% of the 
entire image, there would be an immediate corresponding saving in the bandwidth 
required to transmit the user face (compared to transmitting the entire image)"). 

Regarding claim 19, Gokturk teaches that the step of processing the received 
captured multimedia information to create a graphic comprises: reducing the resolution 
of the captured multimedia information (see paragraph [0077]; "Since, presumably, the 
background content is either stationary or not very useful (in certain applications), it can 
be transmitted with a coarse resolution or/and less frequent (i.e. lower frame rate) 
compared to the foreground segment"). 
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Regarding claim 20, Gokturk teaches that the step of processing the received 
captured multimedia information to create a graphic comprises: selecting a frame from a 
plurality of frames of the captured multimedia information (see paragraph [0079]; "the 
camera captures the user image. By performing an image-processing algorithm, either 
on the local computer or a connected server (by first sending the image to server), a 
computer program analyses the facial impression of the user"). 

Claims 21 and 22 recite a system having substantially the same limitations as 
the system of claims 1 and 2, respectively. Therefore, the claims are rejected under the 
same rationale. 

Claims 24 and 25 recite a method having substantially the same limitations as 
the system of claims 1 and 2, respectively. Therefore, the claims are rejected under the 
same rationale. 

Claims 27 and 28 recite a system having substantially the same limitations as 
the system of claims 1 and 2, respectively. Therefore, the claims are rejected under the 
same rationale. 

Claim Rejections - 35 USC § 103 

The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 
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Claims 5, 12, and 30-34 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Gokturk (2003/0235341) supra and Day et al. (hereinafter Day), 
United States Patent 7,039,676. 

Regarding claim 5, Gokturk teaches every limitation of claim 5 but does not 
explicitly disclose that the trigger is one of a button press, a camera selection, a voice 
command, and a gesture. Day teaches using a gesture to trigger an action in an instant 
messaging application (see Day column 9 lines 16-32; "Upon determination that a 
gesturing event occurred 208... The automatic gesture software would then access an 
associative mapping, e.g., a database of gestures, and find the corresponding action 
and parameter of the action, i.e., the content to be transmitted, for the gesture and the 
state of the gesture 210... For example, "announce" action may equate to the execution 
of "send this string across the network to other participants.'"). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to transmit 
an expression to other chat room participants upon performing a gesture as taught by 
Day in the invention of Gokturk in order to save bandwidth (bandwidth is wasted if the 
same expression is continuously transmitted to every participant). 

Claim 12 recites a method having substantially the same limitations as the 
system of claim 5. Therefore, claim 12 is rejected under the same rationale. 

Regarding claim 30, Gokturk teaches every limitation of claim 30 except that 
said graphic represents motion by said user. However, Day teaches a system that 
analyzes user motion to determine graphics to insert into real-time instant messages 
(see Day column 3 line 36 through column 4 line 2; "The system, method and program 
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of the invention automatically generates input into chat room software that represents 
an actual physical gesture made by a participant in a real time communication over a 
network, such as a "live" chat session or an instant messaging communication"). It 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to use the motion and gesture analysis system of generating emoticons of Day to 
generate the custom icons in the invention of Gokturk for the purpose of making it 
easier for users to generate custom emoticon graphics for transmitting over instant 
message sessions. 

Regarding claim 31, Gokturk/Day teaches that said trigger is a gesture by said 
user and said graphic is a representation of something other than said gesture (see Day 
column 3 line 36 through column 4 line 2; "A video camera, utilized in connection with 
the participants' computer system, captures the real time gestures made by the 
participant, such as a wave, a shoulder shrug, a nodding of the head, and inputs the 
captured video images into the computer system of the participant", receiving a gesture 
is the action that triggers the system to generate an emoticon graphic; see also Day 
Figure 3, showing that the graphic is not a direct representation of the gesture). 

Regarding claim 32, Gokturk/Day teaches that said relevant information 
extracted by said information extraction and interpretation module is in a non graphic 
format (see Day column 8 lines 16-26; "since the configuration process has defined 
each gesture with a corresponding action, a database or table 300 (FIG. 3) becomes 
populated with the gesturing events 301, state of gesture 302, and corresponding action 
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303 and parameter of the action 304, i.e., the content to be transmitted for the gesturing 
event"). 

Regarding claim 33, Gokturk/Day teaches that said relevant information 
extracted by said information extraction and interpretation module is mapped to one of a 
preselected group of graphics, including graphics representing a smile, a frown and a 
wink (see Day column 7 lines 48-58; "the automatic gesture software may provide a set 
of available gestures, e.g., wave hand, smile, frown, wink, shrug, nod, for which the user 
may designate the action (announce, insert text, insert graphic,) and the parameter of 
the action (e.g., the content or translation of the gesture)"). 

Regarding claim 34, Gokturk/Day teaches that said relevant information 
extracted by said information extraction and interpretation module is an article worn by 
said user (see Day column 6 lines 15-32; "The imaging software analyzes various 
features of a participant from captured video frames generated by video camera 115. 
For example, the imaging software may discern any one or more of the following 
features including, but not limited to, the head, eyes, mouth (lips), shoulders, arms, and 
hands. For example, the imaging software can detect whether the head nods up and 
down in successive frames, or if there is a prolonged "wink" in one eye, or if the mouth 
makes a smile or frown, or if the shoulders "shrug", or if an arm or hand moves across 
the captured video frames such as in depicting a wave or other gesture" , the imaging 
software described by Day is capable of detecting and interpreting articles worn by a 
user). 
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Response to Arguments 

Claim 4 has been amended to correct a minor informality. Accordingly, the 
objection to claim 4 is withdrawn. 



Applicant asserts that Gokturk does not teach real time processing to create a 
graphic. The examiner respectfully disagrees. 

Gokturk paragraph [0074] states, "Camera 810 captures the user image in real- 
time... To conserve bandwidth, the server may choose to stream the video of select 
users among the participants in the forum (e.g. the person who is typing)." Gokturk's 
invention captures and processes the user image in real-time and streams it to other 
participants of an instant messaging application. Although predefined images are used 
to create the graphic stream, real time processing is still used to create a graphic, as 
recited in the claims. 



Applicant further asserts that Gokturk does not teach the claimed user trigger. 
The examiner respectfully disagrees. 

Gokturk paragraph [0079] states, "Using the current invention, one can design an 
IM application where the user's facial impression is translated to an avatar and 
transmitted as part of the user message". In the context of an instant messaging 
application as known in the art, this implies that the user's facial impression is only sent 
when the user message is transmitted. In this case, sending the message is the trigger. 
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Gokturk further states in paragraph [0074], "To conserve bandwidth, the server 
may choose to stream the video of select users among the participants in the forum 
(e.g. the person who is typing)." This passage explicitly teaches sending the video 
stream of the user only while the person is typing. In this case, typing is the trigger. 

Conclusion 

The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure: 

• Smith et al. (US 5,870,138), Facial image processing 

• Danker et al. (US 2002/01 84309), Systems and methods for interfacing 
with a user in instant messaging 

• Janakiraman et al. (US 2005/0069852), Translating emotion to braille, 
emoticons and other special symbols 

• Yomoda (US 2005/0078804), Apparatus and method for communication 

• Van Stuivenberg et al. (US 2006/0252455), Multimedia communication 
device to capture and insert a multimedia sample 

Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See M PEP 
§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 
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A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Stephen Alvesteffer whose telephone number is 
(571)270-1295. The examiner can normally be reached on Monday-Friday 9:30AM- 
6:00PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, William Bashore can be reached on (571)272-4088. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

Stephen Alvesteffer 

Examiner 

Art Unit 2175 

IS. A./ 

Examiner, Art Unit 2175 

/WILLIAM L. BASHORE/ 

Supervisory Patent Examiner, Art Unit 2175 



